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ABSTRACT 

A workplace literacy project was conducted at the 
Nabisco Richmond Facility for the following purposes; to determine 
what academic skills are necessary to perform 10 selected jobs, to 
develop a validated assessment to evaluate an employee’s literacy 
skills, to develop a job“specific curriculum to improve skill 
deficits, and to provide ah individualized education program so that 
each employee can gain the skills needed for personal and job goals. 
Approximately 100 employees volunteered for assessment and training. 
During the project, a listening learning module was developed and 
pilot tested and modules on reading and mathematics were developed. 
Because of delays in the project, however, employees did not complete 
the training at the time of project completion. The materials 
developed are continuing to be implemented at the plant, to an 
enthusiastic response from employees and a decrease in waste, 
defects, and absenteeism that can be partially attributed to the 
program. Outside evaluation also showed the program to be a success. 
(The report includes the external evaluation report, a validated 
assessment of job skills, and materials for the listening learning 
module.) (KC) 
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PROJECT NAME: 
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Richmond Enhanced Academics for 

CHANGE 



FINAL REPORT 

National Workplace Literacy Grant Program 
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Program Narrative 



The manufacturing environment at the Nabisco Richmond facility is rapidly 
changing. The value of the traditional skills of the workforce has been mitigated 
by a combination of extensive investment in equipment technology, a program of 
worker empowerment, or Total Quality Control, and a work redesign program. 
This Facility recognizes the need for lifelong learning to keep up with this ever- 
changing work environment. 

In the past, the Richmond Facility utilized a decision making hierarchy typical of 
American industry. That structure placed decision making responsibility and 
authority in the hands of management. Management was entrusted with the 
responsibility to determine the cause and effect of manufacturing inefficiencies, 
and devise action plans for correction. Line workers executed these plans. 
Knowledge of the technology and equipment was prerequisite for a manager but 
not for the general plant employee. Managers needed to be able to comprehend 
equipment and technical manuals; general plant employees did not. Managers 
needed to be able to collect data and apply their knowledge of basic math to 
perform statistical analyses; the general plant employee did not. Managers, not 
the general plant employee, needed to be able to understand manufacturing 
reports and the calculations of standards on which performance measures are 
based. 

This traditional approach of decision making in business is now obsolete. Facing 
challenging global and domestic markets, Nabisco must now utilize the 
collective intelligence of its entire workforce. The continued success of Nabisco 
is dependent on its ability to respond to this changing environment just as the 
continued productivity of Nabisco’s workforce is also dependent on the 
enhancement and progression of employees' literacy skills. 



Recognizing the need for improving the basic academic needs of employees, the 
Nabisco Ricfimond Facility has maintained an on-site Adult Education Program 
as part of the Training and Education Center since 1987. This center offers 
basic Adult Education courses, but until the onset of this project, the generic 
materials have informally correlated to the various jobs in the facility. 

The collaborators in this project are Capital Area Training Consortium, 
Educational Partner, providing curriculum development and instruction. Hay 
Management Group, performing job/task analysis and developing a validated 
assessment tool, and third, the Nabisco Richmond Facility whose employees 
and Union members provide expertise, guidance, and enthusiastic participation 
in the project. Dr. Nancy Berger, Training for Performance, Inc., completed an 
external evaluation of the total project. Her report is attached. 

The purpose of the project is to; 

• determine specifically what academic skills are necessary to perform 
selected jobs efficiently, safely, and accurately. 

• develop a validated assessment protocol to evaluate an employee’s literacy 
skills 

• develop a job-specific curriculum to improve skill deficits 

• give employees the opportunity to self-direct job changes within Union 
agreements 

• provide counseling to employees as to which jobs require the individual’s 
existing literacy skills and 

• provide an individualized educational program which provides each 
employee with the literacy skills needed and sought for personal and job 
goals. 

Therefore this project includes the following 
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• A job/task analysis of ten critical jobs of hourly employees 

• A validated literacy skills assessment of these ten jobs 

• A test / pilot of literacy skills assessment of hourly employees 

• An adult learner involved curricula development 

• A test / pilot of workplace literacy instruction 

• An external evaluation of validated assessment instrument and curricula 

The ultimate outcome of this project is to produce an educational program that 
will ensure that the total workforce, existing as well as future hires, will either 
possess the necessary functional literacy skills or will be encouraged to actively 
pursue these skills using materials that address pertinent job-related knowledge 
but with emphasis on literacy competence. The critical jobs identified for this 
project are Assembler, Mixer, Machine Captain, Baker, Processor, Machine 
Operator, Packing Tech, Utility, Sanitor, Electrician, and Mechanic 

The Richmond Facility is a union environment and the REACH Project operated 
within all contractual agreements between the Company and the Union. 
Establishing a Project Advisory Committee (PAC) including representation from 
management, supervision, union, and REACH staff, ensured all decisions 
related to scheduling, volunteer participation, and confidentiality were within 
compliance guidelines. 

All participation in the project was and continues to be on a voluntary basis. No 
names or existing identification numbers were used to identify each participant. 
A new number was assigned to each employee involved based simply on when 
they started their involvement; therefore the first volunteer was designated as 1 , 
the fiftieth as 50, and so on. Only the Project Director and instructors have 
access to the identity of participants. All staff members signed confidentiality 
statements ensuring complete privacy No individual employee's personal data 
was shared with management or Personnel, 



Program Chronology 



Chronological Summary: 

Scheduled Start Date: March 1, 1993 

Actual Start Date: March 30, 1994 

End Date: September 30, 1 994 

Extended End Date November 30, 1994 

Extended End Date: August 30, 1995 



July 8, 1992 

Grant Proposal submitted to U.S. Dept, of Education 

February, 1993 

The HRD Manager for the Nabisco facility receives a phone call from the U.S. 
Dept, of Education with notification of tentative awarding of grant monies if 
budget and timeline can be negotiated. 

February - April, 1993 

Negotiations continue. Project Director provides detailed timeline per the DOE’s 
request. 

April, 1993 

Project Director and Capital Area Training Consortium, the Education Partner, 
attend Project Directors’ Opening Conference in Washington, DC, April 28-30, 
1993. The official notification of the grant award, with a start date of March 1, 
1993, received at the Richmond Facility on April 27, 1993. 

May, 1993 

Project Director meets with facility/Union Steering Committee May 13, to present 
full grant details and outline Union role. Project Advisory Committee (PAC) 
formed. PAC consists of two hourly Union members. Union Business Agent or 
designee. Project Director, one supervisor, and the HRD manager. Procedures 
for recruitment of participants are defined. 

A video explaining the details of the project is planned. May 19, Union agrees to 
participate in video production. The PAC and Steering Committees meet May 
26. The acronym REACH, Richmond Enhanced Academics for Change, is 
selected for the project’s title. 

Election results for new local Union representatives are posted May 23 New 
Union reps are encouraged to support REACH. 



June, 1993 

International Union meets with facility management June 8. REACH project 
discussed. The consulting firm selected to perform the task analysis is 
disapproved by Union due to a conflict in a non-Nabisco facility. The consulting 
firm had been contracted to perform task analysis and assessment development 
for 10 facility jobs. Union members withdraw from video. Project Director and 
Management continue to produce video. Mass Meetings to kickoff the Project 
scheduled for June 9. The Facility notified June 8 that Union pulls all support for 
the project due to solidarity support of Union issues at another facility and 
disapproval of selected consulting firm. 

July, 1993 

Dept, of Educ. informed of Union concerns. Sarah Newcomb responded that the 
monies were allotted for 18 months and work out a settlement if possible. 

August, 1993 

First Quarterly Report of grant budget submitted to Dept, of Education. 

External Evaluator, Dr. Nancy Berger informed by phone of delay. 

September, 1993 

Project on hold 

October - November, 1993 

Solicitations go out to seek replacement firm for the consulting firm in 
contention. Union agrees to return support if another company is selected to 
complete job/academic task analysis and develop an assessment protocol. 

December, 1993 

Representatives from firms bidding for the REACH contract meet at the facility 
December 13. The purpose and expected outcomes of the project are outlined 
and questions addressed. 

January, 1994 

Project Director meets with representatives of bidding firms separately and 
answers additional questions. Hay Management Group selected to replace 
original consulting firm. International Representative for BCT dies and his 
replacement has to be brought up to date on all aspects of the project. His 
approval must be won. 

February, 1994 

Hay Management Group meets with International Union in Pennsylvania, 
February 17, to discuss Hay's understanding of the project and address any 
Union concerns on job elimination associated with this project. There is and 
never was an issue of job elimination. All aspects of adverse impact are 



considered in all phases of the project development. ADA concerns are also 
addressed. Union approves of the Hay Group and notifies facility of its support. 

March, 1994 

Fliers are distributed explaining the REACH Project. Mass Meetings held March 
30. A statement of support, signed by both International Union and Nabisco 
Corporate, is distributed to facility employees. The Bakery Confectionery and 
Tobacco Workers’ Union (BCTW) Local 358 Business Agent reads the letter of 
support at the Mass Meetings. 

April - May, 1994 

Volunteer pari^cipants are vigorously recruited by posting fliers and holding Q & 
A sessions about the purpose and expected outcomes of the project. Shop 
Stewards attend Q & A sessions and report back to constituents their support of 
the program. Capital Area Training Consortium interviews and hires for 
positions of Curriculum Developer and Lead Instructor. Hay Group sent samples 
of all written materials used throughout the facility. The first REACH Newsletter 
is distributed. 



June, 1994 

Volunteer Internal Consultants, also referred to as SME's or Subject Matter 
Experts, meet with the Hay Group and begin the task analysis process. Subject 
Matter Experts from the jobs of Assembler, Mixer, Machine Captain, Baker, 
Processor, Machine Operator, Packing Tech, Utility, Sanitor, Electrician, and 
Mechanic escort Hay Group reps through their respective jobs, providing 
detailed task analysis with emphasis on what reading, math, graph 
comprehension, problem solving, and listening comprehension skills are 
necessary for each task. 

The Internal Consultants prioritize the tasks by level of difficulty and frequency 
of use. 

It is determined that lo avoid any references to grade equivalents, the levels of 
Minimum, Moderate, and Maximum will be used to categorize skill 
requirements. 

Each group of Internal Consultants establishes benchmarks as references for 
the Min, Mod, Max skill levels. 

All jobs are then cross referenced so the ratings of Min, Mod, and Max are 
consistent and indicative of the level of difficulty for all tasks for all jobs. So if a 
Packing Tech scored Moderate in Reading and a Baker scored Moderate in 
Reading, they are functioning on the same level of reading comprehension. 

Internal Consultants meet with CATC to provide information for curriculum 
development. Different learning styles and teaching modalities are identified. 
Samples of materials requiring the identified academic skills are gathered and 



the uses of these materials are demonstrated by the internal consultants on their 
respective jobs. 

July, 1994 

The Hay Group develops the assessment protocol to assess the Reading 
Comprehension, Applied Math, Graph Comprehension, Listening 
Comprehension, and Problem Solving skills necessary for all jobs/tasks 
identified. 

Using the task analysis material developed by the Hay Group, CATC begins the 
development of curriculum to teach the academic skills for each task. It is 
decided to cluster the skill levels so curriculum to teach to each level covers all 
jobs. So for example, someone wishing to increase academic skills for a 
particular job may be taught with information specific to another job in the facility, 
but the academic skill is the same. This grouping decreases the time and cost of 
developing curriculum for five academic areas, three levels each, for ten 
different job classifications, or 150 different curriculums. Since the academic 
analysis demonstrated much overlapping of skills, to develop 150 curriculums 
would result in redundancy; therefore five major curriculums, each with 
components teaching to all three levels, utilizing diverse job related materials 
resulted. With technical cross training becoming a possibility in the near future, 
using diverse teaching material and resources will expose students to the varied 
materials they may be expected to use in the near future. 

The second REACH Newsletter is distributed. 

August, 1994 

Internal Consultants pilot test the assessment battery. Sessions are timed and 
compared to the estimated amount of time necessary to complete each test. 
Adjustments are made based on the groups’ average completion times. 

Individual test questions are discussed and some rewritten for clarity, face 
validity, and technical accuracy. 

CATC continues curriculum development. 

September, 1994 

International Union Representative meets with the local Union, Management, 
and Project Director for project update. Task analysis and drafts of the 
assessment tool are reviewed. Hourly employees speak of their experiences 
with and support of the project. The International Union Representative 
reiterated the International’s total support of this program designed to keep 
employees job ready. 

CATC continues curriculum development. 



October, 1994 

REACH Newsletter #3 distributcid. Project Director rigorously recruits additional 
volunteers necessary to validate the final assessment tool. She advertises the 
project by posting fliers, meeting employees in informal Q &A sessions in locker 
rooms, walking the production lines, and speaking to the various training classes 
being held in the facility’s Training and Education Center. 

The facility holds a health fair and REACH is represented. A “How is Your 
Academic Health?" quiz is distributed with emphasis on the skills identified as 
necessary for future job needs. Regardless of how well one completes the 1 0 
questions, the final scoring scale recommends a "dose of REACH" to prevent 
future “discomfort on the job.” 

Project Director works with the labor scheduler and supervisors to develop a 
complicated labor schedule which accommodates production mandates, 
vacations, lay-offs, and REACH assessment validation participation. 

CATC continues curriculum development. 

November, 1994 

The first group of employees complete the full assessment battery. Eighty 
employees from all departments except Maintenance and Repair, are relieved 
from regular job responsibilities for the period of testing. Several scenarios for 
testing are tried. Some employees are relieved in ^vo hour increments for four 
to five days, completing one section of the battery daily. Some employees are 
relieved for two half days, while other employees are relieved for one full 8-hour 
shift. How employees were scheduled is dictated by their job classification and 
the ability of providing adequate job coverage in their absence. The employees 
are surveyed as to their preference and the full 8-hour shift, away from all job 
distractions, proves to be the most convenient and accommodating. However, 
production schedules always take priority. 

Each participant critiques assessment battery for clarity, overall job relevance, 
clearness of directions, and allotted completion times. 

All participation is confidential. Each participant is assigned a number, 1 - 102, 
as each enters the testing center. Thereafter, they sign all documentation with 
their assigned numbers rather than a name or .oayroll clock number. If they wish 
to receive the results of the testing, they provide an address to which the results 
are mailed. The letter explaining the results is mailed to “Confidential to 
Assessee # ?" at the address provided. Only the Project Director has access to 
whom each number is assigned. 

Each participant fills out a demographics profile as to gender, race, years on 
current job, and total years employed at this facility. This data, matched with the 
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appropriate identification number is compiled by the Hay Group for Adverse 
Impact and applicable validation analysis. 

CATC continues curriculum development. 

December, 1994 

CATC curriculum developer shadows employees and collect data samples. 
Project Director scores assessments. Each of the five sections of the battery is 
approximately 10 pages. All questions are fill in the blank or essay and 
therefore is tedious and laborious to score. Recruitment for additional 
volunteers continues. 

January, 1995 

Twenty-five new volunteers, half of which are from Maintenance and Repair, are 
recruited to complete assessment for validation purposes. Due to scheduling 
conflicts, only 22 are available to complete testing, bringing the validation total to 
102 employees. Project Director continues the scoring process of the 
assessment batteries. 



February, 1995 

Validation testing is complete. All assessments are scored and the resulting 
data is entered into a computer database. Final test results are mailed. 
Individuals call for appointments to discuss test results and are advised as to 
ho\ to enroll in classes where appropriate. 

Hay starts the validation analysis to use the assessment INTERNALLY. To use 
this protocol for new-hire selection or at other Nabisco Bakeries will require a 
separate ongoing validation. 

REACH Newsletter# 4 is under development. 

CATC continues curriculum development. Slides and videos of job tasks, 
materials, and equipment corresponding to lesson activities are developed. 

March. 1995 

Class schedules are being developed. Registration for classes scheduled for 
week of March 27 - 31 . Classes start within two weeks of registration. Students’ 
enrollment will be staggered based on starting levels and length of time 
estimated for completion of -urriculum per subject area. The Reading 
Comprehension Curriculum is scheduled to be completed by March 31, and 
therefore the first students will be enrolled in Reading only. Students will be 
enrolled in the remaining 4 subject areas as those curricula are completed. 

The curriculum will be critiqued by the students, just as the assessment tool was 
reviewed. Comments on clarity, job relevance, level of difficulty, and overall 
presentation will be acknowledged and the curriculum will be fine-tuned or 
revised when appropriate 
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CATC agrees to start recruitment procedures for instructors. 



April - May. 1995 

CATC and Project Director adjust timeline for curriculum completion. New 
timeline dictates Listening Module will be taught first followed by Applied Math. 
Change made in response to greater need as determined by assessment 
results. Curriculum development contiiiues. 

June. 1995 

Listening Module taught in two-week segments of time. Due to scheduling 
constraints and available curriculum, Curriculum Coordinator and Project 
Director discuss dedicating June and July to pilot group instruction of Listening 
and Problem Solving and August to individualized instruction for Applied Math, 
Graph Comprehension, and Reading. 



July. 1995 

Problem Solving Module taught. 

August, 1995 

Reading, Math, and Graph curriculum shared with pilot group. 

September- December. 1995 

Final reports developed. External Evaluator interviews REACH participants and 
submits report. Curriculum inspected to remove proprietary information. 
Curriculum refined and issues in external evaluation addressed and corrected 
where possible. 



Program Findings 



Included in this final report are reports from Hay Management Group and Capital 
Area Training Consortium defining how the task analysis was completed, how 
the assessment tool was developed and validated, and how the curriculum was 
developed. This was an eighteen month project and the majority of the time was 
spent in task analysis and assessment validation followed by curriculum 
development. The curriculum addresses all academic requirements as defined 
during the correlation of job tasks and required literacy skills but insufficient time 
was available to fully implement the curriculum during the grant period. 

However, pilot groups participated in curriculum sessions for all five subject 
areas and their feedback is denoted in the attached final curriculum report. The 
curriculum should not be considered complete, since it will continue to grow and 
adapt as technology mandates cha. ige. The REACH curriculum has replaced 
the generic material being taught in the Training and Education Center. 
Employee involvement in curriculum development is ongoing. 

As part of ihe grant proposal, correlations between literacy skills and job 
performance would be tracked. Many factors come into play at this point. 
Employees are not assigned exclusively to one job so job tasks change 
frequently. Having supervisors evaluate how academic skill deficits affect job 
performance conflicts with the employee’s confidentiality. Also, supervisors are 
inclined to evaluate an employee on IF a job is completed, not HOW the job is 
completed. Employees with deficits are masters at hiding these deficiencies and 
often receive assistance from co-workers. The supervisor is unaware of the 
“behind the scenes” activity and relates only that the employee reports to work 
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on time and corfipletes jobs as assigned. Separate from this grant, the 
Richmond Facility has investigated the possibility of training supervisors to 
recognize academic skill deficits nonintrusively and confidentially. 



As in any workplace education program the question of how does such a 
program affect the bottom line of the company is considered. Can one prove 
with tangible results that improving basic literacy skills has direct correlation to 
job performance and productivity? Logic dictates that a better informed, better 
educated employee possessing critical thinking skills is better equipped to work 
in an environment now requiring higher levels of academic skills. In an attempt 
to track tangible outcomes, data on the following attributes affecting the “bottom 
line" were collected: waste, rejects, attendance, product quality, safety, 
efficiency and yield. 

When correlating the impact of literacy skills to these measures many factors 
must be considered. Although surveys of employees’ attitudes about feeling 
more competent on the job after instruction corroborate the assumed logical 
impact of enhanced academic skills training, still other factors must be 
examined. 

This facility is experiencing a metamorphosis with many conditions interacting. 
To say that literacy skills enhancement alone affects the improvement of these 
measurements may be deemed arbitrary; however the enhancement of these 
skills DOES play a major role. The foundation of all production processes can 
be summarized in one acronym, POG, or Process Operating Guidelines. 
Employees must possess math skills to accurately take required measurements, 
perform basic statistical computations, and record results in graph format. They 
must possess critical thinking skills to analyze the results and make informed 
decisions when necessary to adjust the process. Further, they must possess 
communication and reading skills to relay this information to and from co- 
workers during the process as well as in summary meetings to discuss progress 
over a period of time. As basic skills are improved, the data collected is more 
accurate and reliable. This synopsis is proven as improvement occurs in each 



of the areas listed above. The best example of this phenomenon is the 
measurement procedure for tracking Product Quality. 

Product Quality is measured statistically. For each product produced, various 
attributes throughout the production process are identified and measurable 
targets for each attribute are established. For instance, the target for the weight 
of a sample of cookies may be 16 ounces. Some variation from this target is 
acceptable and upper and lower control limits allow for this variation. However, 
too much variation is not acceptable and the employee must determine the 
cause of the variation, make a decision on how to correct the situation, and 
report this action appropriately, either verbally, in written format, or by computer 
data input. This data is ultimately analyzed and a final measurement, or Cpk, 
results. 

The Cpk measurement reports that not only is the process under control, but 
how well the process is running to target. Cpk is a comparison of how the 
process is p inning to how the process is expected to run. A Cpk value of 1 or 
higher is desirable. For 1994, the Cpk value foi overall Product Quality at the 
Richmond Facility was 1.03. This value increased to 1.1 for 1995, entitling this 
Facility to earn Nabisco’s President’s Cup, an award to the Facility 
demonstrating the highest levels of Quality as compared with other Nabisco 
Facilities. 

One must consider that to maintain the control as explained above, other 
influencing factors must be taken into account. For example machinery must be 
in good working order, ingredients received must be of highest quality, and no 
abrupt seasonal temperature deviation occurs causing an adverse impact on the 
desired outcome. However, without the employee's accurate data collection, 
analysis, and decision making, this improvement in quality would not occur. The 
skills critical for these tasks are found throughout the REACH curriculum. 
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Annual records track the Facility’s overall efficiency rate. Driven by accurate 
data collection and appropriate monitoring and action taking, the POG process 
is the foundation upon which continuous improvement depends. A formula 
called Theoretical Max is used to determine how well the Facility uses all 
incoming ingredients to produce an expected amount of product. After 
determining how many units of product 'can be produced by this facility, actual 
production numbers are compared. The 1994 efficiency rate was 84.5% 
whereas 1995’s rate improved to 88.4%. 

Some product, for various reasons, does not meet Nabisco’s high quality 
standards and is rejected. By monitoring the process, utilizing Process 
Operating Guidelines, less product is rejected and production efficiency 
improves. The Richmond Facility improved its reject rate in 1995 by 40.8% over 
1994’s rate. 

Another example is the improvement in waste management. Starting 
simultaneously with the REACH Project was the War on Waste in the Richmond 
Facility. All employees were “recruited" to analyze where they could reduce 
waste in their own areas as well as the entire facility. Employees were 
encouraged to submit their ideas for improvement in written format to the WOW 
committee. Employees were made more aware of how to save and recycle. 

Also initiated were production line task forces for employees to identify causes of 
waste and devise ways to decrease it. Employees also participate on Vendor 
Partnership teams. Much of the waste incurred is the result of faulty packaging 
and handling of incoming ingredients. These teams monitor the quality of 
vendors’ products to ensure quality in Nabisco products. Strong reading, 
listening, critical thinking, and communication skills are necessary to function as 
a team member. As a result, suggestions were follov'ed and the 1995 waste 
report showed a ,9% improvement over 1994. 



Another factor affecting job performance is an employee’s attendance rate. An 
employee that feels competent in the job at hand has one less reason to avoid 
work. In early 1994, a dramatic change occurred in how absenteeism is 
monitored and disciplined at the Richmond Facility. The REACH curriculum was 
still in development at the that time and therefore any impact on improvement in 
attendance rates must be accredited to 'the new absenteeism program. 

However, the 2.55% absenteeism rate for 1994 was reduced significantly to 
1.21% in 1995. 

Safety is another area greatly affected by an employee’s ability to listen, read, 
compute, think critically, and follow directions. The Reading Module of the 
REACH Curriculum contains many references to safety guidelines in this facility. 
As a direct result of one employee’s participation in the Reading Module, she 
broadened her vocabulary and better understood the jeopardy she was placing 
herself in by misreading Good Manufacturing Procedures. Overall, the Facility 
showed improvement in the number of accidents per 100 employees over 1994. 
The lost-time accident rate also showed improvement. However, the severity 
rate of the accidents in 1995 was higher than 1994. Of note, one of these 
accidents was directly caused by poor literacy skills, the inability to fully read 
and comprehend a set of instructions. Of interest, this employee was scheduled 
for a Reading class on this very topic the week following the accident. 

In summary, many factors attributed to the overall improvement in the areas 
discussed above. Publicizing the concept of "REACH for your future" and 
stressing Nabisco as an Education Company imprinted the value of lifelong 
learning on the minds of all employees. Providing a program, developed under 
the guidance of co-workers, enabled employees to start the process of seeking 
to improve literacy skills. If the project had a longer time period to continue to 



develop curriculum, test the curriculum, and to document REACH participants’ 
job performance, more evidence would be available to prove direct correlation. 

As the project continues, data will continue to be collected and further 
correlations to job performance documented. Curriculum will continue to be 
developed with an increase in employee input in its development. Outside of the 
Richmond Facility, others will benefit from this project. The curriculum and the 
process followed for this project is being made available to the Virginia Adult 
Basic Education Resource Center as well as to the Centers identified in the 
original grant. Presentations have been made throughout the duration of this 
grant period addressing the purpose and expected outcomes of the project. 



Workshops were facilitated at both the American Association for Adult and 
Continuing Education and the Virginia Association for Adult and Continuing 
Education explaining the task analysis and literacy correlation process. One 
round table discussion was dedicated to the possible problems a company / 
union / grant partnership may incur and ways to develop amicable, win-win 
solutions. 

In summary, the outcome of this project is successful. Overcoming numerous 
obstacles as explained in the Program Chronology, the project persevered. A 
thorough task analysis was completed. Correlating literacy skills were identified. 
A validated assessment tool was developed and a comprehensive curriculum 
addressing job-specific literacy skills was developed and implemented. Pilot 
groups tested and evaluated both the assessment tool and the curriculum 
making recommendations. Over 100 employees volunteered and additional 
employees continue to enroll. 



This project is the collaborative effort of many people, most importantly, the 
employees of the Richmond Facility. Without their continued support and 
enthusiasm a less comprehensive project would have resulted. The success of 
this project ensures that employees at the Nabisco Richmond Facility will 
continue to REACH into the future better prepared than ever before. 



This report has been provided by 
Dr. Nancy O. Berger, Training for 
Performance Plus. 
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SUMMATIVE EVALUATION REPORT 
Nabisco Biscuit Company 
Richmond, Virginia Bi.kery 
REACH Project 



I. Introduction 

Since its inception, the REACH project has incorporated a plan for evaluation of 
the project’s results. After my selection as the project evaluator, I was invited to 
meet members of the Hay Group and Capitol Area Training Consultants (CATC) 
and sit in on early project meetings held by the Hay. Since then I have followed 
the progress of the project through periodic updates. The following evaluation 
report summarizes my observations concerning: the processes used, the content of 
the final project, and the results of the project. 



IT. Qualifications of Evaluator 




Dr. Nancy 0. Berger is a member of the graduate faculty of the George 
Washington University Department of Education and Human Services. She 
teaches primarily for the Masters in Human Resource Development program and is 
the director of tlie off-campus masters program in Hampton and Richmond, 
Virginia. 

With sixteen years of experience in technical training design, development, 
implementation and evaluation, as well as organizational development and 
management. Dr. Berger is also the president of Training for i’erformance, Inc. 

Her clients have included Delmarva Power, Virginia Power, Los Alamos National 
Labs, New England Power Services, Lawyers Title Insurance, the United Nations 
Industrial Development Organization (Vienna, Austria), and a variety of Federal, 
state and local government agencies. 



III. Methodology 

A. The main questions to be answered by this evaluation report are; 

• Did the project produce results? 

• What were those results? 

• ^at were the most successful parts of this project? 

• What might be some areas for improvement in the future? 
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B, The research methods used include: 

• interviews, 

• observation, 

• focus group, 

• professional review of materials. 

The questions for the focus group are included in Appendix A. 



IV. Analysis of the Specific Phases of the Project 
A. Job/Task Analysis - the Hay Group 





Description 

The Hay Group undertook a detailed job and task analysis through 
observation and discussion with volunteer employees (internal consultants) 
in the specific jobs. A team of experienced Hay professionals carried out 
this analysis, primarily in June, 1994. The intent of this procedure was to 
determine the main tasks involved in performing various bakery jobs, and 
to determine the underlying academic skills (math, reading, listening, and 
so on) needed to perform those tasks. The job classifications analyzed 
included Assembler, Mixer, Machine Captain, Baker, Processor, Machine 
Operator, Packing Tech, Utility, Sanitor, Electrician and Mechanic. 

After completing the task listing. Hay G’oup and the internal consultants 
ranked each task according to the follo’/^ng criteria: time spent, task 
difficulty, consequences of error, and importance. Basic skill levels were 
rated by Minimum, Moderate, and Maximum. No grade level equivalents 
were used. All jobs were then cross-referenced so that the skill ratings 
(minimum, moderate, maximum) would be consistent for all jobs. 

Evaluator’s Analysis 

The job/task analysis was thorough and professional. The level of detail of 
the task analysis was sufficiently specific for the purposes of the project, 
but not overly detailed. The use of Internal Consultants at all stages of the 
job/task analysis insured a realistic portrayal of the jobs as analyzed and 
interpreted by the Hay Group. 

The Internal Consultants validated the results and, in a later focus group, 
commented very positively about the process and the results. They 
described the process as one of brainstonning the i mportant things that 
employees do, without getting sidetracked by less relevant details. The 
employees clearly expressed respect for the goal orientation of the 
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consultants and their professionalism. They felt that the process was 
efficient, and that time was seldom wasted. They noted that all of the Hay 
consultants knew the process to use and where they were headed, and 
followed through. Comments included: “Pretty thorough.” “Knew what 
they [Hay Group] needed to do.” 

The use of rating levels without reference to grade levels was an excellent 
decision. The terms minimum, moderate and maximum accurately reflect 
skill levels without the negative^ connotations that grade levels might 
contribute. These levels were clearly defined and differentiated. 

B. Development of Assessments 

Description 

Using the information provided by the job/task analysis, the Hay Group 
developed an assessment protocol to measure Reading Comprehension, 
Applied Math, Graph Comprehension, Listening Comprehension, and 
Problem Solving skills needed to perform the tasks. 

The Hay Group and Internal Consultants pilot tested the assessments. They 
measured and adjusted completion times and gathered information about 
specific questions. The Hay Group used statistical analysis to identify 
items tliat were too easy or too hard and items that showed bias by gender 
or race. As a result, a number of items were eliminated or revised.. The 
adjustments and clarifications can be seen when comparing pilot 
assessments and current forms. 

Evaluator’s Analysis 

The assessments are clearly based on the job/task analysis data. Valid 
statistical processes were used, and these were thoroughly explained so that 
it was easy to see what had been done - this documentation will also be 
helpful for future reference and use. 

The focus group participants mentioned the tapes used in the listening 
portion as an example of portions of the assessment that needed 
improvement. However, they also noted that the REACH project team was 
very responsive to their feedback and adjusted the assessment accordingly. 
The content of the revised assessments includes a fair amou nt of bakery- 
related material. The partnership of experienced professionals from the 
Hay group and Internal Consultants from the bakery provided a balance of 
expertise which is reflected in the job/task analysis and assessment 
development and revision. Fiinployees expressed a strong sense of being a 
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part of the development of the program, a critical element in a project of 
this type. This employee “buy-in” should help insure continued interest in 
the program. 

C. Curriculum Development - General 
Description 

Based on the data gathered by the Hay Group, the Capital Area Training 
Consortium developed clusters to streamline and simplify the development 
of the curriculum. The color coded mapping of skills and min/mod/max 
levels helps illustrate and clarify the clusters. This method provides a good 
visual overview of the scope of project and a good reference document. 

D. Program Content - Listening Skills 
Evaluator*s Analysis 

The Listening Skills portion of the program involves a rich variety of 
exercises. These encoinage learner involvement and active listening skills. 
Beyond listening skills, this section also includes more general 
communication skills. For example, the learner participates in following 
directions, but also in giving directions. The section also addresses issues 
such as workplace cultures, group dynamics, and group decision-making. 
While this content may be helpful, the section title, “Listening Skills,” is 
somewhat misleading. It is, more accurately, a section devoted to 
communication and group interaction skills, as well as listening. 

The focus group stated that they did not like some of the subjects on the 
tapes, which made listening more difficult. They suggested that the tapes 
would be better if more directly related to job issues or to the bakery. They 
also found the tapes confusing at times. Beyond the tapes, however, they 
emphasized the value to them of working in groups, sharing job information 
and discussing job-related items. They enjoyed learning about other jobs 
and issues in the plant. While this may not have been the intended learning, 
it would certainly seem to be a beneficial by-product. 

According to participants, the main benefit of this section was developing 
an awareness of the need to listen and of different perceptions when people 
“hear” things differently. They mentioned that they found themselves 
listening more closely outside of work, as well, such as at church and at 
home. 
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E. Program Content: Reading 

The focus group indicated that the reading comprehension section met their 
expectations for that section. They commented that seeing other points of 
view, and realizing that there were no right or wrong answers, was the best 
part. Their main criticism involved items not related to work. As one 
individual stated, “If you can’t get into the reading, you don’t care.” For 
example, they liked the reading about Oreos, but were not interested in the 
article about donuts. They emphasized that it would be better if all the 
readings were job/bakery-related. 

The portion dealing with words was “great”. The focus group spoke 
enthusiastically about learning about other parts of the plant and what other 
employees do. They noted in particular that they more readily understand 
what others are talking about as a result of learning new vocabulary. This 
portion of the learning was clearly helpful to them. 

F. Program Content: Applied Math 

Despite the perception that many people dislike math, the focus group 
expressed satisfaction with this portion of the learning. They mentioned 
that there was a good variety of problems, from easy to hard. The math was 
a good review, “especially of things you knew but don’t use much” and 
may have forgotten. They noted diat this section was “very thorough,” and 
found it very relevant to the job. They particularly appreciated the review 
of decimals, especially dividing decimals, and of percentages. In some 
cases the computer does the math for them, but they like knowing the math 
steps. As a side benefit, several noted that they are now more able to help 
their children with their homework. 

The group felt that they could have used more time in this section, and 
would have liked more explanation of how to do problems in some cases. 
They found that instructors were very helpful and gave good directions. 

I hey seemed to take special pride in having learned and “conquered” the 
math section. 

G. Program Content: Graphs 

According to the focus group the most valuable result of this section was 
recognizing different types of graphs and better understanding what they 
show. They noted that they feel more comfortable in meetings now, 
because they understand more about the visual graphs used. This section 
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served as a good refresher, since the computer does much of the work that 
used to be done manually. The participants enjoyed the group discussion, 
and mentioned the value of one-on-one assistance from the instructor. They 
also noted the side benefit of better understanding graphs in the newspaper, 
magazines, on television, and so on. 

H. Program Content: Problem-Solving 

The entire group expressed their enjoyment of this segment. They learned 
about how hard it is to come up with one “right” answer, and found it 
interesting to see different approaches and ideas. It was a “great learning 
experience.” They learned to listen to everyone’s ideas and recognize that 
there may be different ways to reach the same valid result. They found that 
it has helped them in their work with others on the job. 



General Comments on the Curriculum and the Project - Participant Viewpoint 

From the participants’ point of view the program is valuable to them both at work 

and off the job. When speaking of the program as a whole, they mentioned: 

• Made tliem realize how much they had forgotten. 

• Just felt good about the whole thing. Better self-esteem, felt that it made 
him or her a “better person.” 

• Wonderful instructors. Knew their material, were down to earth, explained 
well, took time to work with individuals, patient, attitude made you want to 
attend and participate. 

• Never lost interest throughout the project. 

• Felt involved throughout the project. 

• One individual said that the program itself brought him/her out, speaks out 
more, and more comfortable doing so. 

• Got to know other people in other departments and appreciate their jobs and 
viewpoints -- You don’t tend to blame other people so much if you 
understand their side. 

• Learned a lot and “just enjoyed the program.” 

• Need to change with the times, and programs like this help. 

• The project newsletter was a good w'ay to keep people informed. 
Communications are essential i. a project of this type so that everyone 
knows what is happening, and why. Good communications also help 
prevent misunderstandings, and feelings of “what are they trying to hide?” 

I he group expressed few criticisms. They did point out that it is hard to do the 

program on overtime, because of family schedules and other conflicts. It would he 







easier to focus on learning and attend if done on company time. Several also felt 
that the program could be lengthened, and that they needed more time. They felt 
that a great deal was compressed into short times frames in some cases. 



VI. General Comments on the Curriculum and the Project - Evaluator Viewpoint 

From a professional vantage point, I find that the project was very successful. A 
great deal of hard work went into producing a program that would be relevant, 
well-accepted, and valuable to participants. In any project of this scope, there are 
“ups” and “downs”. The project team experienced union challenges, tig’ t 
deadlines and other difficulties, but was able to keep the project moving. 

From the begiiming the project plan was clear and followed established steps for 
the creation of a program of this kind. The Hay Group provided efficient, 
professional support in developing the job/task analysis and assessment. CATC 
produced a solid curriculum built on the foundation laid by the Hay Group. The 
volunteers who served as Internal Consultants were highly motivated and 
enthusiastic throughout. 




VII. Recommendations for Improvements and for the Continuing Quality of the 
Program 

The following are recommendations for maintaining and improving the quality of 
the REACH program. 



A. Upgrade the quality of the curriculum materials. 




1. The materials are adequate, but could be more appealing. With 
simple desktop publishing additions, the materials would appear 
more professional. Add page numbers and some illustrations. Use 
graphics to enhance the appearance and readability. Some minor 
improvements would turn good materials into great materials. A 
professional format sends a signal to the employee that he or she is 
important to the organization and deserves the best materials. The 
current version is a good final “draft”. 

2. Improve the quality of articles, clippings and otlier “paste-ups”. 
Some of these look somewhat “thrown together”. Also improve 
copy quality. Some of the pages I received were not clear. If 
necessaiy', retype articles and other items so that they can be read 
more easily -- some older individuals need more accessible text. 
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3, ' Try to incorporate even more Nabisco-related examples, problems, 

and exercises. In comparison to many basic education programs, the 
current curriculum is somewhat customized for the bakery. 

However, the focus group voiced a strong desire for even more of 
this customization. 

4. Avoid whole, monotonous pages of math problems. These look 
downright forbidding, and are overly reminiscent of school. From 
my research in adult learning with similar participants, I found that it 
is cnicial to avoid reminding participants of earlier school 
experiences, which may not have always been pleasant. Break 
problems up, come up with more word problems, or otherwise 
change these sections. 

B. Support the continuation of this project. 

Several individuals expressed concern that the company would not be 
willing to continue support for a project of this kind. With the level of 
enthusiasm and interest in the program as evidenced by many of the 
participants, discontinuing the project would cause a loss of morale. 

C. Recognize employee concerns about other uses of the program. 

Several employees voiced concern that the company would eventually 
expand the use of this program to qualify or disqualify individuals for jobs 
and promotions. They also seem unsure about the future confidentiality of 
the program. From their perspective the company has made promises in the 
past that it has not kept, and they do not totally trust the organization’s 
motivations. Whether accurate or not, this perception indicates a lack of 
trust. It is crucial for future dealings with employees that they see a more 
positive support and confirmation from the organization. 



VIII. Constraints to the Evaluation 




The individuals involved in the project from Nabisco and Hay were very open and 
cooperative in the evaluation of this project. The main constraint to the research 
was the imavailability of the actual participant evaluation forms. These would 
have provided valuable insight into the participants’ reactions to the program at the 
time they participated. However, their comments in the focus group did provide 
their evaluation of the project with the added value of distance from the 
experience. This distance helped them weigh the value of their Icaniing as it 
actually applies to work and home experiences. 
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IX. Summary 



# 

Overall, the project was well-planned, well-managed, and professionally executed. 
The Project Leader, Internal Consultants, External Consultants and others involved 
in the project should be congratulated for hard work that has resulted in important 
benefits to the individual learners and to Nabisco’s operations. With some fme- 
tuning a very good program will be excellent. 



# 

i 




O 

ERIC 



-10 



32 



Appendix A 



Focus Group Question Outline 
September 20, 1995 



1. Welcome and Introductions 

* Who I am 

* Confidentiality 

* Who are you - Name, involvement with Reach 

* How we will proceed 

* What I vnll do vnth the results 



2 . 




Job Analysis - (June/July 1994) 

* How was the job analysis done? 

* What was your perception of the process? 

* Did the consultants accurately and completely analyze the main tasks 
of the jobs? 

* Did the job analysis provide a clear picture of the jobs? 

* How did the consultants rate the tasks as minimum, moderate, 
maximum skill requirements? Did you agree with their ratings? 



3. Assessment Pilot - (Aug. 1994) 

* Were you involved in the pilot of the assessments? 

* How was that conducted? 

* Did you recommend changes? Do you know if those changes were 
made? 



4. Final Assessment - (Nov. 1994) 

* Did you take the final assessment? 

* Did your assessment results fit your perceptions of your abilities? 

* Were you confident that the results were kept confidential? 



5. 




Actual Materials 

Who participated in each of the following parts of the program? 
Reading Comprehension 
Listening 
Applied Math 
(iraph Comprehension 
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Problem Solving 



What were your expectations of what you would learn from this part 
of the program? 

Did this part of the program meet your expectations? 

What was the best thing about this part of the program? 

How was the information taught? 

Could you easily see the link between the learning exercises and 
skills you use on your job? Were the materials job-related? 

What difTiCulties did you have going through the exercises, 
materials? 

What suggestions would you have for improving the exercises, 
materials? 

Have you seen any improvement in your job performance as a result 
of the learning activities? 



6. Overall Program Comments 



What did you get out of the project personally? 

- Has the program helped you at work? 

- Has the program helped yo u outsid e of work? 

What do you see as the value of this program to Nabisco? 

Do you have an overall sense of confidentiality about your 
participation and results? 

How do you feel about the program overall? Would you recommend 
it to a co-worker? Was it a positive experience? 

What suggestions do you have for improvements? 
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Development of the Nabisco Basic Skills Curriculum 



Purpose 

The purpose of this report is to document the approach to the development of the curriculum and 
to report the results of the limited instructional activities that were designed to evaluate the 
effectiveness of what had been developed. The report also addresses the challenges and 
opportunities associated with the effort given the abbreviated time line for the work to be done, and 
provides recommendations for continuation and further development of the instructional program. 

Development of the Curriculum 

The basic skills curriculum was based on the work of HayGroup in conjunction with subject matter 
experts from the Nabisco Bakery, Richmond, Virginia. The HayGroup was responsible for 
completing a task analysis on the ten job types found in the Richmond Bakery. The analysis 
produced a task list of 750 tasks that were then analyzed according to the basic skills required to 
complete the tasks. The required performance level of each of the task was also indicated for each 
task. 

The HayGroup also established benchmarks for each job type that retlected the complexity of the 
basic skills required to perform each task. Skill level requirements were defined as none, minimal, 
moderate, and maximal. 

Based on the job analysis information, the HayGroup developed assessments for five types of basic 
literacy skills; Reading Comprehension, Listening, Applied Math, Graph Comprehension, and 
Problem-Solving. 

The HayGroup completed the task analysis in June of 1994 and the draft assessment tool in 
September of 1994. Curriculum development planning began in May of 1994 without the final task 
analysis and without any validation of the assessment tool that would ultimately direct inuividual 
workers into appropriate basic literacy skills training designed to meet their individual objectives 
with respect to their current job or the job or jobs they aspire to secure in the future. 

The Capital Area Training Consortium was contracted with to develop a set of instructional modules 
that would facilitate an individualized basic skills development plan and instruction in the areas 
identified by the assessment process as not meeting expected performance levels for the job type 
selected by the worker. 

The initial work involved exploration of the best approach to consider given the individualized 
nature of the training and the restriction imposed by the project of not teaching any job specilic skills 
that were non-literacy in nature. We decided to examine the 750 tasks and chose to create clusters 
of related tasks. This resulted in the creation of 51 clusters I'f related tasks. 



We then examined the skills and skill levels contained in each of the 51 clusters. The HayGroup 
facilitated our efforts in this regard by providing us with a run on the 51 clusters we identified 
incorporating the 750 tasks. From this analysis, we were able to gen i ate a grid that reflected basic 
skills across job types, the highest skill level needs for each skill in the 51 clusters (See Attachment 
A). We color coded the desired skill levels for successful job performance. We applied the 
HayGroup definition of minimal, moderate, and maximal to the color coding, We also used the 
HayGroup's benchmarks developed for each skill level and researched the various strategies that we 
might employ to move into the design of the modules. 

We then developed flexible, topical outlines for the five skill areas that would be used to develop 
the curriculum content by skill level. The outlines we designed to be used like a menu in that they 
were comprehensive for each skill type across job types and applicable to any skill cluster. 

This approach vvas compatible with the assessment instrument that was under development and 
resulted in the five modules reflecting the five literacy skills identified and analyzed by the 
HayGroup. It reflected the observabii. behaviors and activities required to demonstrate successful 
job performance at each of the three skill levels, minimal, moderate, and maximal, and included 
inclusion of the thinking skills required to perform at each of the three skill levels. 

Although the original objective was to create individualized instruction for each basic literacy skill, 
observation of the workplace, input from the subject matter exoerts and a recognition that the 
workplace required interaction among and between various workers in getting and giving 
information and in solving work related problems as part of a team, we opted to develop the 
Listening and Problem-Solving modules as grc)up activities. The remaining three modules. Applied 
Math, Graph Comprehension and Reading Comprehension were developed to be individual self- 
paced instruction. 

Curriculum Development Challenges 

Inadequate Development Time Allotted 

Prior to the completion of the assessment battery or a draft of the assessment tool and prior to its 
administration and reporting of results, the decision was made by the Nabisco Project Coordinator 
to develop curriculum based on the demographics co^ined in the proposal accepted ly the U.S. 
Department of Education who funded the project. T^proposal indicates that the majority of the 
workers were functioning on a 6th grade reading level and that they lacked the math skills needed 
to perform the required work task at the very basic lov/est level. This is where we began our work. 
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On May 26, 1995, we received the results of the assessment administered bv the Nabisco REACH 
Program Coordinator. The assessment revealed that the initial demographics did not align with the 
skills reflected by the volunteers who were a part of the validation study. The majority of those 
tested fell into the "minimal" level in Listening and Problem-Solving, not Math and Reading as 
anticipated. In fact, reading and math were moderate and above. Armed with this new information, 
we approached the REACH Coordinator and agreed to alter the order of curriculum development. 

This allowed us thirty days in which to develop the curriculum and to begin instruction as required 
by the REACH Coordinator. This also provided the same thirty days to hire instructors that were 
to become familiar with the curriculum and prepared to provide instruction by June 26, 1995. 

One of the most difficult aspects of the curriculum development process was the lack of 
adequate time to fully develop each module to its most desirable level prior to beginning the 
instruction or adequately preparing the teaching staff prior to its implementation. The 
curriculum development process and the implementation of the instructional were originally 
contracted as an 18 month effort that was limited to 13 months as the beginning stages of the 
project bogged down. 

Gathering of Workplace Materials 

The materials encountered and used by workers on the job in the performance of the ten job 
categories was not assembled prior to the beginning of the curriculum development process. It 
would have been efficient and more useful to have the materials available by job type for use in 
developing curriculum. Having the materials available, assembled, labeled, organized and 
inventoried (to ensure that proprietary and confidential materials remain so) would have greatly 
facilitated a smoother transition from the task analysis to the curriculum development process. 1 his 
preparation and planning would have allowed the company and its training staff to weed out 
materials that were out of date, irrelevant, proprietary, or confidential, as well as having permitted 
a better assessment of the readability level of the materials. 

Not having materials readily available and organized in advance by the company greatly hampered 
and delayed curriculum development efforts. 

Manpower Shortage 

Once the project timeline became abbreviated, additional curriculum developers would have made 
the process and the product better. Initially, a full time developer and a halftime instructor were 
assigned to the project in anticipation of an 18 month development and instructional cycle. I his 
proved to be inadequate as the project loss five months of development time. More emphasis and 
manpower devoted to curriculum development that was not separate and distinctly removed from 
the instructional activities would have produced a more comprehensive package and allowed more 
thorough development. 
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We also suffered from an inability to attract and retain instructors for the three shifts required to meet 
the grant requirements. Losing instructors meant taking time away from the curriculum development 
to cover classes and to retrain new staff on the curriculum to be taught, 

Administrotive Concerns 

As the project moved forward, it became clear that more formal communications would have aided 
in the clarification of the concerns and needs of the company with respect to managing the 
participants. Marketing of the REACH program, communication with supervisors of the workers 
who volunteered for the program, and the soliciting and selection of volunieers were all done by the 
REACH Coordinator. Sharing of the infomiation in writing that was provided to supervisors, and 
the rub s established by the company for volunteer participation, and clarification of the role of the 
REACH Coordinator, with respect to ensuring adherence to any requirements imposed by the 
company, would have lessened the involvement of the Instructional staff and the Curriculum 
Developer in such matters. Instructors, for example, were asked to be responsible for passing oui 
paychecks, but were given no instructions on how' to handle checks for workers who did not attend 
the class that day. Monitoring attendance and other matters that were administrative were 
incorporated in the day to day activities. 

Problems ranged from participants not being informed about their schedule for attending to a 
participant's check not being available when other participants received theirs in class. These matters 
eroded time that could have been spent on instruction. 

Instructional Challenges 

The purpose of the assessment battery was to measure an individual's skill level. Based on the 
worker’s individual skill level, appropriate recommendations were to be made regarding refresher 
or academic training and education. For example, if an individual was assessed as having a 
minimum level of applied math skills and a moderate or maximum level of applied math skills is 
required for the existing or desired job, the individual could choose to: take no action to develop his 
or her skills, pursue math education in the community, or enroll in a Math course offered by Nabisco 
at the bakery. This purpose statement was extracted from the HayGroup's report on the development 
and validation of the assessment battery. 

Given this purpose for the assessment, the purpose of the curriculum developed b\’ the Capital Area 
Training Consortium was to provide the on-site ability to improve in those areas that failed to meet 
desired skill levels. 

As the instruction began, individuals were included in curricula that may not have been appropriate 
because their assessment results may not have warranted participation. I'or example, individuals 
who achieved a maximum skill level of achievement on the listening assessment may have been 
assigned to the listening module by the REACH Coordinator, I'he challenge was ensuring that the 
time for the individual w;is not wasted nor considered none valuable. 
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Scheduling of employees was also a real challenge. Forty hour blocks of time were ^et aside for 
various groups of employees to participate in instruction across three shifts. Vacations, leave and 
other interruptions caused some individuals to not receive the full forty hours of instruction. Erratic 
attendance was a challenge that may or may not be overcome. Such attendance detracted from the 
ability to secure adequate evaluation and feed back on the curriculum. 

Participant Evaluations of the Curriculum 

REACH Volunteers who completed the Listening Module and those who completed the Problem- 
Solving Modules were the only groups that were able to provide feedback that should be used to aiter 
the curriculum. The Curriculum as presented to Nabisco on August 31, 1995 does not reflect the 
many revisions that should be made in response to the evaluations, nor does it reflect the many 
alterations and revisions the curriculum developer would like to see made. The inadequate time 
frame for completion of the effort means that the Company will need to build on the curriculum 
delivered on August 31, 1995 to reach the ultimate objectives stated in the proposal to the U. S. 
Department of Education. 

The participants evaluations of the Listening and Problem- Solving curriculum were positive. They 
also offered excellent recommendations for change and noted concerns that should be addressed in 
the activities and time allotted for instruction. .A copy of their evaluations and the summaries 
prepared by the instructors may be found in Attachment B. 

Recommendations and Conclusions 

Recommendation # I Continue .he Instruction to Allow Adequate Evaluation of the Curriculum 

Given that the REACH Project did not develop as outlined in the proposal submitted to the U. S. 
Department of Education. The opportunity to develop curricula and the opportunity to test curricula 
via instruction and the opportunity to evaluate skills transfened is a significant problem. 



In the grant, it is stated that the partners will evaluate the effectiveness of the assessment and 
subsequent customized curricula by three methods. First, the partners were to track the qualification 
rate of the job applicants (j^’b bidders) post program implementation verses pre-program 
implementation. The enhanced ability of the employee to qualify for career advancement would 
reflect success in the project. The second evaluation method was to be a statistical evaluation ot the 
correlation between enhanced literacy skills and the key manufacturing indicators of waste, rework, 
rejects, down time, yield, and labor efficiency. Comparative date for individuals, work teams, 
departments and the overall plant in general were to be used. The third method was to depend on 
input from the adult learner him/lierself. Using a questionnaire and interviews, the responses would 
then be evaluated. 
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Obviously the evaluation plan will be altered to correspond to the work that can be evaluated given 
the loss of five months of project time. Absent the time to use the evaluation information, no 
changes have been made to the first issue of the curriculum. 

In recognition of the need to fully test and evaluate the curriculum, the Capital Area Training 
Consortium began work with the company through Tom Currin before his departure to locate 
additional funds to permit the in-house REACH Coordinator to continue instruction. A 
S10,000 request was made to the Henrico IDA for funding through .June 30, 1996 for three 
shifts of instruction. No final decision has been made by the company. 

Recommendation #2 Continued Development and Use of Individual Educational Plans 

The Capital Area Training Consortium developed the Individual Educational Plan (lEP) as a vehicle 
for trainee and instructor to keep focus on the objectives for participation (Attachment C). The lEP 
should reflect the assessment results and allow the trainee to establish the skill levels they would like 
to achieve, given their desire to improve current job performance or to qualify for other job 
opportunities within the bakery. The final format and the need for continued confidentiality should 
be finalized to be a grid that reflects the assessment results and the objectives established in each of 
the five basic skill areas. As with the one tested during the project, the time and attendance should 
be recorded on the back as a way to establish the number of hours invested for the success 
e.xperienced. This can be a very valuable piece of data for on going evaluation of the curriculum and 
the instruction. 

Recommendation Scheduling 

Although the Capital Area Training Consortium was not involved in scheduling employees for 
participation, the scheduling during work hours was very problematic given the essential nature of 
production and needed manpower. We strongly suggest that consideration be given to having the 
program available, but during the non-scheduled work hours of employees. This would prevent 
sporadic attendance caused by shifting work schedules. 

The curriculum was required to be delivered in forty hour blocks with each participant being given 
5 consecutive 8 hour days off the work floor for all instruction. This i' lot the most effective 
instructional method. While this approach works for the group activities, it may prove too intense 
for all participants given the subject and the skills development involved. If the program truly 
intends to be an individualized self-paced activity, then individuals should be able to schedule 
themselves into the hours and times that they are able to commit to. If the program were held during 
non-work hours, then this would be more realistic. 
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Recommendation n4 Expand Curriculum Content 



Although the project restricted our ability to teach any of the job specific functions not specifically 
related to the literacy needs, adults learn best by having the ability to transfer learning immediately 
while doing. The disadvantage that an isolated literacy training activity has is its apparent 
disconnect from the other activities of the workplace. In this project, the REACH program would 
have significantly benefited from being a part of a larger organizational effort to retool its workforce. 
It was unclear to the participants, if not the project designer, that it was in fact valued by the 
company and its employees. It may have been helpful to have the literacy skills development 
incorporate some aspect of the technical skills required for the jobs held or being sought. There is 
a whole body of knowledge that supports this approach. The Capital Area Training Consortium was 
prohibited from pursuing any inclusion of the technical skills. 



Conclusions 




We believe that our approach to the curriculum is a sound one given the task analysis and the 
subsequent development and validation of the assessment battery. Wliat is needed is a review of the 
assessment battery' to ensure that any alterations made as a result ot the validation process and that 
any changes in the skill levels and associated activities are captured in the cumculum properly. The 
curriculum package is only the first step in the need to now refine and revise, based on participant 
feedback and on the instructors experience with delivering instruction. More activities and varied 
practice opportunities need to be developed, as well as feed back from the subject matter experts who 
contributed to the process as it was developing to refine the activities. 



The curriculum is in its infancy and can be grown to meet the needs of the individual workers and 
the workplace as the evaluation process takes place. 
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REACH Listening Skills 
Module Evaluation 



REACH Listening Skills Module Evaluation 



Instruction for the Listening Skills Module began on June 27 and concluded on 
July 7. Thirty four (34) individuals participated in two-hour class sessions for 
eight days, resulting in 16 hours of instruction. See the Statistics and Comments 
Section for attrition details. 

Facilitators guided the participants through group discussions, role playing, 
cassette tape listening, and workbook activities. Nineteen (19) activities of 
varying skill levels were completed. On the last day of instruction, the 
participants were asked to evaluate the course content and effectiveness by 
completing a questionnaire (see attached questionnaire). Overall, the participants 
rated the course as "excellent, well organized, and effective." The module 
objectives were clear, and many commented that they had improved in some areas 
and learned new listening techniques. The workbook activities were described as 
"good and relevant." Participants felt not enough time was devoted to some 
workbook activities. A weakness noted by a majority of the students is the 
module's inability to teach them how to handle situations when supervisors refuse 
to adhere to listening skills guidelines. However, the course was not structured to 
deal with management's skill level or personnel issues. 



Major Conclusions/Recommendations 



Strong Points Noted by Participants 

• The listening skills training was beneficial. 

• The module was well organized. 

• The objectives were made clear to participants. 

• The workbook activities were relevant to jobs. 

Weak Points Noted by Participants 

• There was not enough time was allotted for some exercises. 

• Some participants are not sure the listening skills guidelines can be used 
on a daily basis in the plant because manager’s do not practice the 
guidelines. 

• Some participants wanted more practice using active listening techniques 
in groups and with supervisors. 

Recommendations Noted by Participants 

• Require all plant personnel to take listening skills review courses, twice a 
year at least. 

• Managers and supervisors should take REACH Listening Course. 

• Allow more time for activities in the workbook. 

• Do not give homework assignments. Provide a way to do telephone 
listening activity in class. 

• Make adjustments to several of the activities or delete them. 
Recommendations Noted by Facilitators 

• Delete some activities that were not effective in encouraging interaction. 

• Shorten some activities to allow adequate time for other activities. 

• Revise several of the activity sheets in the notebook. 

• Do not schedule students during their vacations or days-oft. 

• Remind students that the course cannot deal with personnel problems. 

• Provide students with snacks (some were disgruntled due to lack of sodas, 
donuts etc.). 
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Statistics and Comments 



Listening Skills Module Participants 



Session 

Time 


Number 

Scheduled 


Number 

Added 


Number 

Deleted 


No 

Shows 


Actually 

Attended 


Number 

Attritions 


Number 

Transfers 


7:30- 

9:30 

a.m. 


18 


0 


4 


5 


7 


2 


0 


1:00- 

3:00 

p.m. 


18 


0 


J 


2 


7 


5 


1 


3:30- 

5:30 

p.m. 


12 


3 


4 


1 


5 


5 


0 


9:00 - 
11:00 
p.m. 


4 


1 


1 


1 


2 


1 


0 


Totals 52 


4 


12 


9 


21 


13 


1 



Legend: 

Attrition = Those who began the class, but did not complete the class 
because of vacation, days-ofT, or personal circumstances. 

Transfers = Those who transfen'ed from one class to another. 

Actually Attended = Those w'ho attended all class sessions. 

Conclusion: The attrition rate was very high due to scheduling and other 
reasons. A total of 34 (21 +13) individuals participated in the 
class. 
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The chart below represents the percentage of students that rated the listening 
module excellent, good, fair, or poor overall. 
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Number of total students: N =21 

(11 checked excellent; 9 good, 0 fair, and 1 poor) 



Conclusion; Overall, the participants said the calss was excellent. 

The chart below records the skill level of the students that attended the class. 



N/A -Not 
Tested 

Mod 




8 




Min 




6 
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Comments 




The course help make better working environment better for all workers. 

The course helped them me evaluate myself to be more open-minded. 

Helped me understand how to better deal with co-workers. 

Personally motivated me to work on improving my listening skills on the 
job as well as outside of work. 

Group discussions were veiy- helpful and attention getters. 

Should slow down the pace. Some activities went too fast. 

Should allow more time for group listening activities and more practice. 

Clarify directions of activities in the yellow section by numbering. 

I received some additional knowledge from the course that 1 can use at a 
later time. 
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REACH Problem Solving/Critical Thinking Module Evaluation 

Instnjction for the Problem Solving/Critical Thinking Module was conducted during the 
weeks of July 17 through 28, 1995. Thirty (jO) participants attended classes two hours per day 
for a total of 20 instructional hours, The participants represented several different job titles 
including mixers, machine captains, general helpers, packing technicians, machine operators, 
sanitors, and maintenance & repair technicians. Ali of the participants were assessed at the 
minim.um skill level in problem solving. 

They were guided through 19 activities. Many of the activities were group oriented and 
focused on the four basic steps of solving problems. The four steps are common to all problem- 
solving situations. A summary of the four step process follows: 

•Identify the problem - Ask the question, "What's wrong with this situation?" 

•Generate ideas to solve the problem - Ask the question, "What can I do about it?" 

•Try out ideas to fix the problem - Ask the question, "Will this idea fix the problem?" 

•Evaluate the resolution - Ask the question. Will this idea fix the problem?" If the problem 
is not fixed, the process is started again. 

Participants were asked to troubleshoot workplace problems and apply the problem 
solving step process to those situations. In addition, figural analogies were introduced to 
encourage participant's to practice critical thinking and deductive reasoning. On the last day of 
instruction, 17 participants completed the course evaluation questionnaire. The results show that 
participants believed the course objectives were beneficial and the objectives were clear. Many of 
the students stated the troubleshooting process was reinforced by the course materials. 

The materials presented in Part I - (mind map, common elements of problems) and Part 
III (practical application) were given the highest ratings of good and excellent (3=good, 
4=excellent). Part II (analogies, deductive reasoning) received ratings of fair to good (2=fair, 
3=good). Some participants mentioned that they did not clearly understand how figural analogies 
directly related to the workplace resolution process. 

However, many of them commented that the analogies and deductive reasoning materials 
were good. They simply needed more time to get a better grasp of the analogy concept. Overall, 
the course delivery and content was rated as good 



Conclusions and Recommendations 




Strong Points Noted hy Participants 
Course delivery and content were good. 

•The course makes me think of more pu.->;:ive soluticnis to [iroMems and situations 
•Visual aids were helpful 
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•Objeaives were clear; an overall excellent class. 

•The four-step process was very good and helpful. 

•The deductive reasoning activities of addition (math) we. e good. 

•Charts were neat and legible and always available. 

•Good problem solving techniques. 

•The numbers and analogies were the strongest features of the 
course. 

Weak Points Noted by Participants 

Not enough time devoted to understanding the analogies section. 

•Not enough time for class to do some activities. 

•Students from other countries would have problems understanding. 

•The shapes and figures are not ones we come in contact with (quads, pentas). 

•The analogies need to be explained and gone through more slowly. 

Recommendations Noted by Students 
•Provide more time for the analogies activities. 

•Allow all employees to participants in the course. 

•Group activities are productive. Have more of them. 

Recommendations Noted by Facilitator 

•Provide a mind map of a work related problem in conjunction with the mind map of the "Door 
Obstacle" problem. 

•Change the wording in activity number seven to ideiitify the types of problems as simple, 
complicated, and complex. Do not use the wording "rather complicated". 

•Include more examples of problems that may occur in the Sanitation/Utility areas. 

•Provide more information ir * 'e "You Need to Know" section about how figural analogies relate 
to problem solving and critical ' unking. 

•Activities seven and nine seem t ) be repetitious. Combine or rewrite the activities. 

•Many students were not informed of the dates to report to classes. Better communication 
regarding class schedules is necessary. For exam.ple, one student noted that he did not report to 
class because he was not notified by his supervisor. 



Statistics 



Participants of the Problem Solving/Critical Thinking Module 



Class 

Time/ 

Section 


Number 

Scheduled 


Number 

Added 


Number 

Deleted 


Actually 

Attended 


Number 

Attrition 


7:30- 
9:30AM 
Sec. A 


7 


1 


2 


5 




11- 1PM 
Sec. B 


J 


0 


0 


2 


1 


1-3PM 
Sec. C 


5 


0 


J 


j 


0 

1 


1-3 PM 
Sec. D 


5 


1 


1 


6 


0 i 


3:30- 
5:30PM 
Sec. E 


5 


1 


0 


5 


0 1 


3:30-5:30 
Sec. ?■* 


>> 


0 


1 


0 


0 


7-9PM 
Sec. G 


2 


1 


0 


J 


0 


9-1 1PM 
Sec. H 


2 


1 


0 


3 


0 


1 1-7AM 
Sec. I 




0 


0 


J 


0 1 


Totals 


35 


5 


7 


30 


J 




■'Class section was canceled. Two students were transferred to other sections The one remaining 
student was dropped from the class 



Conclusion: The majority of the students completed the course. However, many missed days 
due to vacation, and days-off. Others were not notified by supervisor when to report to class 
For example, one participant in class Section C showed up for the last day of instruction 
Approximately, eight students were tranferred between the class sections because due to 
scheduling conflicts and the cancellation of Section F. Four students never attended the class. 
The participants accounted for under the attrition column did not complete the module because of 
of vacation days or other personal reasons 






Comments (taken from the evaluation questionnaire) 

•The class was very helpful. 

•The facilitato^are excellent teachers. 

•All Nabisco employees should be allow^ed to take this course. 

•Mary Tanner is very informative. 

•Longer time for classes and have the classes to continue weekly. 

•Group study is good. This helps students, especially those from other countries, so they would 
not be intimidated. Group study will enhance these students' understanding. 

•Ms. Walker is an exciting instructor with alot of zeal and part of the now generation group. 

•It was surprising to see some of the things you knew, but were not using. 

•This course helped me to improve my way of reasoning and taught me a new concept of 
problem solving 

•Good problem solving techniques and some of the math kept me on my toes. 

•I learned that in problem solving, sometimes there are so many different answers. 

•I liked ‘he numbers sequence. We don't do math much an>more. People use calculators. 
•Classroom set-up was not desirable for five people. Teacher was in front of the board. 

Prepared by: Kim B. Walker, APR 
July 1995 
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REACH Background and Purpose 



The Nabisco Corporation obtained a grant from the U.S. Department of 
Education to develop an innovative employee training progiam that will be 
used as a model for other corporate training programs around the country. 

The program, entitled Richmond Enhanced Academics for Change (REACH), 
uses bakery materials to enliance the academic skills of employees. The 
REACH curriculum, which was developed by the Capital Area Training 
Consortium staff, includes five modules. Those modules are: Listening, 
Reading, Problem Solving/Critical Thinking, Applied Math, and Graph 
Comprehension. 

Approximately 100 individuals participated in a pre-screening process to 
assess their skill levels. Participants tested at the minimum, moderate or 
maximum skill levels. Instruction for the Listening Skills Module began on 
June 27 and concluded on July 7. Thirty four (34) individuals participated in 
two-hour class sessions for eight days, resulting in 16 hours of instruction. 

See the Statistics and Comments Section for attrition details. 

Facilitators guided the participants through group discussions, role playing, 
cassette tape listening, and workbook activities. Nineteen (19) activities of 
varying skill levels wei e completed. On the last day of instruction, the 
participants were asked to evaluate the course content and effectiveness by 
completing a questionnaire (See attached questionnaire). Overall, the 
participants rated the course as “excellent, well organized, and effective.” 

The module objectives were clear, and many commented that they had 
improved in some areas and learned new listening tecliniques. The workbook 
activities were described as “good and relevant”. Participants felt not enough 
time was devoted to some workbook activities. A weakness noted by a 
majority of the students is the module’s inability to teach them 
how to handle situations when supervisors refuse to adhere to listening skills 
guidelines. However, the course was not structured to deal with 
management’s skill level or personnel issues. 



Major Conclusions/Recommendations 



Strong Points Noted by Participants 

• The listening skills training was beneficial. 

• The module was well organized. 

• The objectives were made clear to participants. 

• The workbook activities were relevant to Jobs. 

Weak Points Noted by Participants 

• There was not enough time was allotted for some exercises. 

• Some participants are not sure the listening skills guidelines can be used 
on a daily basis in the plant because manager’s do not practice the 
guidelines. 

• Some participants wanted more practice using active listening techniques 
in groups and with supervisors. 

Recommendations Noted by Participants 

• Require all plant personnel to take listening skills review courses, twice a 
year at least. 

• Managers and supervisors should take REACH Listening Course. 

• Allow more time for activities in the workbook. 

• Do not give homework assignments. Provide a way to do telephone 
listening activity in class. 

• Make adjustments to several of the activities or delete them. 
Recommendations Noted by Facilitators 

• Delete some activities that were not effective in encouraging interaction. 

• Shorten some activities to allow adequate time for other activities. 

• Revise several of the activity sheets in the notebook. 

• Do not schedule students during their vacations or days-ofT. 

• Remind students that the course cannot deal with personnel problems. 

• Provide students with snacks (some were disgruntled due to lack of sodas, 
donuts etc.). 



Statistics and Comments 



% 

Listening Skills Module Participants 



Session 

Time 


Number 

Scheduled 


Number 

Added 


Number 

Deleted 


No 

Shows 


Actually 

Attended 


Number 

Attritions 


Number 

Transfers 


7:30- 

9:30 

a.m. 


18 


0 


4 


5 


7 


2 


0 


1:00- 

3:00 

p.m. 


18 


0 




2 


7 


5 


1 


3:30- 

5:30 

p.m. 


12 


3 


4 1 


1 


5 


5 


0 


9:00- 

11:00 

p.m. 


4 


1 


1 


1 


2 


1 


0 


Totals 52 


4 


12 


9 


21 


13 


1 




Legend: 

Attrition = Those who began the class, but did not complete the class 
because of vacation, days-off, or personal circumstances. 



Transfers = Those who transferred from one class to another. 

Actually Attended = Those who attended all class sessions. 

Conclusion: The attrition rate was very high due to scheduling and other 
reasons. A total of 34 (21 +13) individuals participated in the 
class. 
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The chart below represents the percentage of students that rated the listening 
module excellent, good, fair, or poor overall. 






Number of total students: N=21 

(1 1 checked excellent; 9 good, 0 fair, and 1 poor) 



Conclusion: Overall, the participants said the calss was excellent. 



The chart below records the skill level of the students that attended the class. 



N/A -Not 
Tested 

Mod 




8 




Min 







0 5 10 



15 




92 



4 




Comments 
















The course help make better working environment better for all workers. 

The course helped them me evaluate myself to be more open-minded. 

Helped me understand how to better deal with co-workers. 

Personally motivated me to work on improving my listening skills on the 
job as well as outside of work. 

Group discussions were very helpful and attention getters. 

Should slow down the pace. Some activities went too fast. 

Should allow more time for group listening activities and more practice. 

Clarify directions of activities in the yellow section by numbering. 

I received some additional knowledge from the course that I can use at a 
later time. 
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Development and Internal Validation 
of the Nabisco Basic Skill 
Assessment Battery 

Purpose 

The puipose of this report is to document the results of tlie Pilot Test of the Basic Skill 
Assessment instruments developed for the Richmond baker)'. The report describes how 
the assessments were developed, the purpose of the analyses performed for the validation, 
their results, and the impact of the analytical findings on the final form and potential use 
of the assessments. 




Development of Assessment Battery 



The assessment battery was developed by the HayGroup in conjunction with job . 
incumbents from the Nabisco Bakery, Richmond, Virginia during the summer of 1994. 
For its development, several steps were taken. First, the project was introduced to bakery 
employees via the employee newsletter, posted flyers, informal discussion with the 
Project Coordinator, Ms. Judy Bailey-Kinker, and formal project kickoff meetings with 
the HayGroup. After the project's introduction, four Hay consultants made job site visits 
to the Richmond bakery during the first, second, and third shifts. The job site visits had 
several purposes: for the HayGroup consultant to become familiar with the bakery 

environment and jirotocol, talk to job incumbents about their daily activities, and observe 
job documentation ^e.g., training manuals, safety notices). The consultants observed and 
talked with incumbents from ten job titles: Assembler, Baker, Electrician, Machine 

Captain, Machine Operator, Mechanic, Mixer, Packing Technician, Sanitor, and EHS 
Utility. 



From the information gathered during the site visits, a preliminar)' task list was 
developed for each of the ten jobs. A series of meetings were then held with job 
incumbents to review and edit the task information and identify the type of basic skill and 
level of skill required for successful performance of each task. Basic skills included in 
the analysis were: reading comprehension, listening, applied math, graph comprehension, 
and problem solving. Skill level requirements were defined as: none, minimal. 
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moderate, and maximal. To identify such information, job incumbents were given real- 
world examples of each type of skill at each level. The incumbents were asked to 
generate an example for each skill level that was job-related. The result of these meetings 
was a detailed description of the type of basic skills required of a job incumbent for task 
performance, as well as the level of skill required. 

Based on the job analysis information, separate assessments were written for five 
types of basic literacy skills: 

• Reading comprehension 

• Listening 

• Applied math — 

• Graph comprehension 

• Problem solving 

Items in each assessment were written to simulate the type of information or calculations 
encountered in a job or real-life situation. 

The purpose of the assessment battery is to measure an individual's skill level. 
Based on his or her skill level, appropriate recommendations can be made regarding 
refresher or academic training and education. For example, if an individual is assessed as 
having a niiiiimum level of applied math skills and a moderate or maximal level of 
applied math skills is required for the desired job, the individual can choose to: take no 
action to develop his or her skills, pursue math education in the community, or eruoll in a 
Math course offered by Nabisco at the bakery. 



Description of Assessment Batter\' 

The assessment battery is a paper-and-pencil test composed of 5 parts: Reading 
Comprehension, Listening, Applied Math, Graph Comprehension, and Problem Solving. 
When appropriate, practice items or examples are provided. A detailed description of 
each part is as follows. 

Reading Comprehension 

The Reading Comprehension assessment contains seven passages (e.g., a 
television schedule, advertisement, recipe, safety information) and three to five 
questions for each. The assessee is told that the passages require the 
understanding of written English sentences and paragraphs. Assessees should 
read the passage zmd answer the questions given. This assessment should take 
approximately 30 minutes, but no more than 45 minutes, to complete. 

Listening 

The Listening assessment is designed to measure the assessee's ability to 
listen and to comprehend verbal statements. The assessee is asked to listen to a 
25- minute tape recording. The recording contains seven passages or scenarios 
(e.g., completing a grid, setting a clock) with two to six questions for each. The 
tape is played only once. While listening to the tape, the assessee may take notes 
on the answer sheet that is provided and then answer the questions. This 
assessment should take approximately 40 minutes, but no more than 55 minutes to 
complete. 

A pplied Math 

The Applied Math assessment is designed to measure the assessee's ability 
to add, subtract, multiply, and divide numbers accurately. Instructions are given 
to read the word problems and/or perform computations, and write the answers in 
the spaces provided. Computations may be performed on the answer sheet and 
scratch paper. The Applied Math assessment should take approximately 30 
minutes, but no more than 45 minutes, to complete. 



Graph Comprehension 



The Graph Comprehension assessment requires the interpretation and 
understanding of graphs and charts. Instructions require the assessee to reviev’ 
the graph or chart and answer several questions that follow. A variety of graphs 
and charts are represented: one point-to-point graph, one pie chart, one table, two 
bar charts, and three line charts. This assessment should take approximately 30 
minutes, but no more than 45 minutes, to complete. 

Problem Solving 

The Problem Solving assessment reuuires the assessee to identify and 
follow rules to solve problems or to arrange items in a particular order. Separate 
instructions are provided for the eleven problem types (e.g., analogies, logical 
thinking, coding, following a map). This assessment should take approximately 
35 minutes, but no more than 50 minutes to complete. 

The assessments are designed to be administered individually or as part of a 
battery. If the assessments are administered as a battery, total testing time without breaks 
is a minimum of 2 hours and 45 minutes. A total of 4 hours should be allotted to 
administer the entire battery. 
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Pilot Test Methodology' 



Data for the analyses described in this report came from a pilot, or evaluation, test of the 
assessment instruments. One hundred bakery employees volunteered to take part in the 
test. The number who completed each aissessment is listed in the table below: 



Assessment 


Number of employees participating 


Reading Comprehension 


99 


Listening 


95 


Applied Mathematics 


100 


Graph Comprehension 


99 


Problem Solving 


100 



The assessments were scored and the results entered into computer files for statistical 
analysis. The passing score for each level (minimal, moderate and maximal) was set at 
correctly answering 80% of the items at that level. 

Statistical Analysis and Recommendations 

Overv'iew 

The statistical analysis of the pilot test data was made up of two major phases. The first 
of these phases focused on tlie evaluation of the basic structure, or psychometric 
properties, of the assessments. The second phase evaluated the assessments in terms of 
their possible adverse impact btised upon minority status or gender. 

Phase I Analyses Results 

The purpose of the Phase I analyses was to determine if the individual items, or 
components, of the assessments were functioning in the proper manner. The key aspects 
of this phase of the analysis included the determination that the items were of the 
appropriate difficulty for their target level, and that they were measuring the same skill 
as the rest of the items of the assessment. 

The first step in answering these questions was to calculate the pass-fail rates (i.e., the 
percentage of individuals ptissing) for all of the items within each assessment. A sample 
of this analysis is displayed in Table 1 (on the next page), and a full copy of the analytic 
output is included tis Appendix A of this report. In these tables, the column labeled 
“Frequency” gives the actual number of pilot test participants who passed or failed the 
item. The column designated “Percent” shows what percentage of the total pilot test 
pauticipants corresponds to that frequency. The percentage of individuals passing the 
item is called the “ptiss-fail rate” for the item. The columns labeled “Valid Percent” and 
“Cum Percent” are automatically produced by the statistical analysis program, but are not 
relevant for the analyses on hand. 

The pass-fail rates for each item were compared to the other items in its assessment level 
(i.e., an item for moderate level reading comprehension was compared to the other 
moderate reading comprehension items). Items identified as too easy or too difficult for 
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their assigned level were placed in a pool for possible elimination or reassignment to a 
different level. 



tClCIK^ 


.lA 










Valid 


Cum 


Value 


LjOMl 




Value 


Frequency 


Fercent 


Fercent 


Fercent 








.00 


49 


49.0 


49.5 


49.5 








1.00 


50 


50.0 


50.5 


100. 0 












1.0 


Idssiag 










Total 


100 


100.0 


100.0 




Valxd 


cases 


99 


ICissiiig cases 1 









minimal level of ability. “O’’ refers to those who incorrectly answered the item, 
and “1” to those who correctly answered the item. 



Table 1. Sample Analysis: Item Pass-Fail Comparison 

The next step in evaluating the fimetioning of the assessments was to determine if the 
items measured the skills they were designed to assess. For example, Hay determined 
that Reading Comprehension item 4A, which asked employees to read an article and 
select the most appropriate title, was not measuring reading comprehension skill. Hay 
also determined that Graph Comprehension item 1, which asked employees to plot a 
series of points on a chart, was not measuring graph comprehension skill. This 
determination was accomplished in part by using correlational analysis to measure the 
relationship between each item and the other items within the assessment level. A sample 
of this analysis is illustrated in Table 2 below, while the full analytic output is contained 
in Appendix B. 







- - Correlation Coefficients 


- - 




BCXZKIAX 


KCliZN_lA 




FCiaNlAX 1.0000 


.2004 






( 99 ) 


( 99) 






P= . 


.005 




BCiaN lA .2>04 


1. 0000 






( 99 ) 


( 99 ) 






9 ^ .005 


. 






Again, the notation is similar to the previous example, RCMIN^IA refers to 
the Reading Assessment item number lA testing for the minimal level of 
ability. However, in this case an additional variable, RCMFNIAX, refers to the 
entire score for Reading Comprehension assessment items testing at the 
minimal level minus the unique contribution of the variable RCMIN_ 1 A 





Tabic 2. Sample Analysis: Item to Assessment Comparison 
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Correlations are expected to fall between zero and one, with high numbers indicating a 
strong relationship. In the example shown in Table 2, the correlation item lA in the 
reading comprehension test to the total score of all of the other items on the minimal level 
subtest is .28. Ninety-nine (99) individuals in the pilot test completed this item and the 
other items in the reading comprehension test. The correlation is statistically significant 
(p=.005), meaning that a correlation this large cannot be attributed to chance or random 
vtiriation. 

Items whose correlations with the rest of the assessment were negative or close to zero 
were placed in a pool for possible modification or elimination. For example, the 
correlation shown in Table 2 of .28 between item RCMTN_1A and the rest of the reading 
comprehension minimal level subtest is lower than one would expect (such correlations 
should be around .8). Therefore, the item (which dealt with reading the time and date on 
a TV schedule) was revised for the fmal assessment battery. 

Phase I Analyses Recommendations 

These two types of item analyses indicated that the assessments were generally 
psychometrically sound. However, they also indicated that the assessments could be 
improved through some minor changes. These changes included: 

1 . Dropping nine items from the assessments, 

2. Moving three items from the “MINIMAL” to the “MODERATE” level, 

3. Moving nine items from the “MODERATE” to the “MINIMAL” level, and 

4. Moving one item from the “MODERATE” level to the “MAXIMAL” level. 



Assessment 


Level 


Total items 


Possible Points 


New Threshold 
80% 


Reading Comprehension 


Min 


15 


15 


12 


Reading Comprehension 


Mod 


7 


7 


5.6 


Reading Comprehension 


Max 


8 


8 


6.4 


Reading Comprehension 


Sum 


30 






Listening 


Min 


5 


5 


4 


Listening 


Mod 


8 


11 


8.8 


Listening 


Max 


7 


12 


9.6 


Listening 


Sum 


20 






Applied Math 


Min 


23 


23 


18.4 


Applied Math 


Mod 


10 


II 


8.8 


Applied Math 


Max 


13 


14 


11.2 


Applied Math 


Sum 


46 






Graph Comprehension 


Min 


7 


7 


5.6 


Graph Comprehension 


Mod 


12 


12 


9.6 


Graph Comprehension 


Max 


8 


10 


8 


Graph Comprehension 


Sum 


27 






Problem Solving 


Min 


1 1 


1 1 


8.8 


Problem Solving 


Mod 


10 


10 


8 


Problem Solving 


Max 


7 


0 


7.2 


Problem Solving 


Sum 


28 







Tabic 3. Final Hem Content for All Assessments 
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Table 3 shows the item content, or design, of the assessments after incorporating the 
changes noted above. It should be noted, that of the 173 items in the original 
assessments, only 22 (or 13%) required revision based upon the psychometric analyses. 

Phase II Analyses Results 

The purpose of the Phase II analyses was to evaluate the assessments for possible adverse 
impact. The key aspect of this phase of the analysis involved comparing the assessment 
pass-fail rates for different sub-group (e.g., minority status and gender) differences. For 
those assessments that showed sub-group differences, additional sub-group comparisons 
were conducted on the pass-fail rates based on the individual items of the assessment. 

Two different ty pes of comparisons were conducted. The gender comparison contrasted 
the passing rate (percentage of the total who passed) of males to the passing rate of 
females. The minority status comparison contrasted the passing rate of whites to the 
passing rate of non-whites. 

The results of these comparisons are summarized in Table 4 below, and a complete 
outpmpf the analyses are included as Appendix C. A sample of the output is illustrated 
in Table .5 (on the next page). In the example illustrated in Table 5, the first number in 
the table’s cells shows how the 63 females and 35 males who took tlie pilot test of 
reading comprehension were spread across the three levels of the assessment (e.g, 5 
females and 3 males were at the minimal level). The second number shows the percentage 
of the 63 females and 35 males who took the pilot test for reading comprehension at each 
of the three levels (e.g, 7.9% of the females and 8.6% of the males were at the minimal 
level). The most important feature of these analyses is the Chi-Square table on the 
bottom. If either the “Pearson” or “Likelihood Ratio” significance level (the last column) 
is less than .05, the analysi.« indicates that tliere are important differences between the 
subgroups (in this case females versus males). The text underneath the Chi-Square tables 
is automatically produced by the statistical analysis program, but are not relevant for the 
analyses on hand. 










Assessment 


Gender Comparison 


Minority Status Compariso 








Reading Comprehension 


No Difference 


No Difference 








Applied Mathematics 


No Difference 


Whites Higher 








Graph Comprehension 


Males Higher 


Whites Higher 








Problem Sohtng 


No Difference 


Whites Higher 








TTslening 


nIo Difference 


^TiTtes Higher 









Tabic 4. Sub-Croup Analyses Summary 
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BEST COPY AVAILABLE 



Several of the assessments showed different pass-fail rates as a function of sub-group. 
Findings of this sort are common for skill assessments' but indicate that using the 
assessments for selection or promotion may result in adverse impact for those minorities. 
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PC Pat tarn (3 lavala) by GENDEE Gandar 






GEKDEE 


Page 


1 of 1 




Count 










Col Pet 


0 


1 


Kow 






F 


M 


Total 


KUlf a Kr< J 


2.00 


5 


3 


8 


PFF 




1.9 


0.4 


8.2 




3.00 


25 


8 


33 


PPF 




39.1 


22.9 


33.7 




4.00 


33 


24 


57 


PPP 




52.4 


48.4 


58.2 




Column 


43 


35 


98 




Total 


44.3 


35.7 


100.0 


Chi 


-Square 




Value DF Significance 


Pearson 






2.91C73 2 .232C2 


Likalihood Patio 




3.01287 2 .22170 


Minimwi Expected Frequency - 


2.857 




Cells with Expected Frequency 


< 5 - 


1 OF 4 ( 14.7<b) 


Kwdier of 


Miaainq Qbaarvationa : 1 




In this example PFF refers to those scoring at the MINIMAL level for the Reading Comprehension assessment; 

PPF refers to those scoring at the MODERATE level for the Reading Comprehension assessment; and 

PPP refers to those scoring at the MAXIMAL level for the Reading Comprehension assessment. The various forms of 

the Chi-Square test are alternative ways of determining the statistical significance of the sub-group comparisons. 



Table 5. Sample Analysis: Passing Rates of Sub-Groups 



Given that some of the assessments showed different pass-fail rates as a function of sub- 
group, the next step was to make similar sub-group comparisons at the item level. The 
purpose of these comparisons was to identify ziny items with large sub-group differences. 
These items were then examined to determine potential sources of the differences A 
sample of this analysis is illustrated in Table 6 (on the next page), while the full analytic 
output is contained in Appendix D. 



' See Reynols, C.R., & Brown, R.T. (Eds ) (1984) Perspective!; on bias in menial tesiinf;. New York: 
Plenum. 
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These analvlic outputs are very similar to those illustrated in Table 5. The difference is 
that the output shows how the 63 females and 35 males who took the pilot test of reading 
comprehension did on a particular item. In this example 30 females and 19 males passed 
the item. Again, the second number shows the same infonnation in a percentage format 
(e.g., 47.6% of the females and 54.3% of the males passed the item). In this case there are 
three analyses presenting the Chi-Square table on the bottom (i.e., “Pearscn,” “Continuity 
Correction,”^ and “Likelihood Ratio”). If the significance level (the last column) of any 
of these statistical tests is less than .05, the analysis indicates that there are important 
differences between the subgroups (in this case females versus males). The text 
underneath the Chi-Square tables is automatically produced by the statistical analysis 
program, but are not relevant for the analyses on hand. 

This analysis identified six items that could be improved through the addition of 
examples. These items involved rounding of numbers and examples were added to 
clarify the rounding procedure desired in each item. These changes may reduce sub- 
group differences in future applications of the assessments. 



KCUlN_lft by GENDER Gender 



GENDER 



RCUIN IR 



Count 
Col Pet 


0 

F 


1 

H 


.00 


33 


16 




52.4 


45.7 


1.00 


30 


19 




47. € 


54.3 


Column 


63 


35 


Total 


64.3 


35.7 



1 of 1 



Sow 

Total 



49 

50.0 

49 

50.0 

98 

100.0 



Mlnlimm Expectod Freqaoncy - 17.500 

Number of IClsexnig Qbeervatxons : 1 



CKi-Square 


Value 


BF 


Sxgnx ficance 


Paaraon 


.48000 


1 


.52709 


CoatxnuitT Correction 


.17778 


1 


.67329 


Lxkallliood Katxo 


.40037 


1 


.52690 



Where RCMIN^IA is the first item in the Reading Comprehension test, 
minimal level. The various forms of the Chi-Square test are alternative ways 
of determining the statistical significance of the sub-group comparisons. 



Table 6. Sample Analysis: Item Sub-Group Comparison 



' The Continuity Correction is only relevant for two by two tables, so it was not output for the previous 
analyses. 
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Phase II Analyses Recommendations 

The primary outcome of the comparison of sub-group differences was that use of the 
assessments for selection or promotion could lead to adverse impact for some groups. 
This finding means that before the assessments are used for that purpose, they must be 
validated against actual job performance. Once this validity is established and 
documented, the assessments can then be used for selection. 
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